idfc-coder RAG Interview Notes

Answer to give

idfc-coder needs to retrieve the right code context before the LLM can act on a ticket, and that retrieval layer is what I designed. A developer gives a JIRA ticket or a question, and the system must fetch the files the LLM needs. We index about 5 lakh (500,000) code chunks. Code needs two kinds of search: semantic search for conceptual questions, and exact keyword search for function names and config keys. So we need hybrid search, which means a store that supports both. We evaluated Qdrant, Cassandra with vector support, and OpenSearch, profiled how long each took, and chose OpenSearch because it does keyword and vector search natively in one system. I would revisit the choice if chunk count or query volume grows well past today's scale, or if our hybrid-search quality needs change.

Points to hit, in order

1. One-line summary first (retrieval feeds the LLM).
2. Scale as a number: 5 lakh chunks.
3. Two search types needed: semantic and exact keyword.
4. Alternatives weighed: Qdrant, Cassandra plus vector, OpenSearch.
5. Decision and the reason: native hybrid search in one system, backed by profiling.
6. Exit criterion: what would make you switch.

Watch out

• Do not say IDFC already runs Postgres. Say pgvector only wins when Postgres already exists.
• Do not trail off with "I don't remember exactly". Give an approximate number and say it is approximate.

Notes box

Q1 shape
 1. One line: retrieval feeds the LLM
 2. Scale: 5 lakh chunks
 3. Two searches: semantic + exact keyword = HYBRID
 4. Weighed: Qdrant, Cassandra+vector, OpenSearch
 5. Chose OpenSearch: both search types natively, profiled
 6. Exit: revisit if scale or hybrid needs grow

Answer to give

I would not treat it as one failure. I would walk the pipeline stage by stage. First, was the right code indexed at all? If not, the gap is in ingestion or chunking. Second, was it in the wide retrieval results, the top 50? If not, it is a retrieval problem, caused by the embedding model, the chunking, or missing keyword search for exact terms. Third, did it survive ranking into the final top 8 to 10 that go to the model? If not, it is a ranking problem. Fourth, if the right context reached the model and the answer was still wrong, it is a generation or prompting issue. Dividing it into stages tells me exactly where to focus instead of guessing.

Notes box

WRONG ANSWER: find the broken stage
 1. Not indexed?          -> ingestion / chunking gap
 2. Indexed, not in top 50? -> retrieval problem
                              (embedding, chunking, no keyword search)
 3. In top 50, not top 8?   -> reranker / ranking problem
 4. Right context, wrong answer? -> generation / prompt issue

 Rule: find which stage dropped it BEFORE blaming the LLM.

Vocabulary

• Call stage 2 a "retrieval problem". Embeddings are only one possible cause.
• Use standard numbers: top 50 for the wide pool, top 8 to 10 for what reaches the model.

Answer to give

Our search needed both semantic and exact keyword matching, because developers search by concept and also by exact identifiers. Qdrant is a strong dedicated vector database, but then keyword search is a second system to run and keep in sync. We also considered Cassandra with a vector layer. We profiled the candidates for latency, and OpenSearch gave us both search types natively in one system within our time budget. The trade-off is that OpenSearch is heavier to operate than a single-purpose store. I would revisit if hybrid quality or scale outgrew it. (Add your real latency number here if you have one.)

Why not pgvector

pgvector wins only when Postgres already exists, because you keep one system and get SQL filtering. Without Postgres, that advantage is gone.

Corrections to remember

• Say "Qdrant" clearly.
• OpenSearch wins because it supports keyword and vector natively together, not because it is open source.
• Say "semantic search and keyword search". Semantic search already is vector search.
• Always name the alternative you weighed before saying why you leaned away from it.

Notes box

CHOOSING WITHOUT EXISTING POSTGRES
 Core question: semantic only, or hybrid?  Code -> HYBRID
 Qdrant       + best vector perf   - separate keyword system to sync
 OpenSearch   + both natively      - heavier to operate
 pgvector     - only wins if Postgres already exists
 Decision: profile on real queries, pick by measured latency/recall

Answer to give

I do not look at RAG as one score. I break it into three layers. First, retrieval: did we find the right code, measured with recall at 50, plus precision and ranking quality for the final context. Second, generation: given the right code, did the model use it faithfully and answer the real question, and does it say "I don't know" when context is missing. Third, cost: not just cost per query, but cost per successfully resolved task, because a cheap system that fails often is the more expensive one. I report all three together, because a good score in one layer can hide a problem in another.

The metrics, in plain words

Cost per resolved task, worked example

System A: 30 per task, 50% success -> 30 / 0.50 = 60 per success
System B: 40 per task, 90% success -> 40 / 0.90 = about 44 per success
B is cheaper per success even though A looks cheaper per attempt.

Notes box

THREE QUESTIONS, NOT ONE SCORE
 1. Did we find it?        -> Recall
 2. Is it ranked / useful? -> Precision + ranking quality
 3. Did the model use it?  -> Faithfulness, relevance, abstention
 COST: per query (basic), per RESOLVED task (the honest number)

Follow-up: how do I build the system that checks quality?

I would not rely on manual spot-checking. I would build a golden dataset from our own Git history: past JIRA tickets as the questions and the files actually changed to fix them as the correct answers. I run it automatically whenever chunking, the embedding model, or retrieval logic changes. When testing an old ticket I use the index as it was before that fix, otherwise the fix is already in the index and the score looks falsely perfect. I still review manually, but only the 20 to 30 worst answers each week.

BUILDING THE EVALUATION SYSTEM
 1. Golden set: ticket text = question, changed files = answer (200-500)
 2. Test against the OLD index (before the fix)
 3. Automate: recall, precision, faithfulness on every change
 4. Weekly manual review of the ~20-30 worst answers, bucketed:
    not indexed / not retrieved / ranked low / model ignored it

Answer to give

We do not re-index the whole codebase on every change, which would not scale. A webhook fires on every push to the main branch and tells our indexing service which files changed. We re-chunk and re-embed only those files and replace their old chunks. This is incremental indexing. For deleted files or removed code, we delete the old chunks, so the index never points to code that no longer exists.

Vocabulary

• Say "webhook" for the trigger and "incremental indexing" for the method.
• Say "vector index" or "retrieval database". Do not say "RackDB".
• Separate the cases: additions and modifications are re-chunked and re-embedded; deletions remove chunks entirely.

KEEPING THE INDEX FRESH
 Trigger: webhook on push to main
 Action:  re-chunk + re-embed ONLY changed files (incremental)
 Deletes: remove chunks of deleted code too

Answer to give

Authentication and authorization at login are not enough. The real control is a metadata filter inside every retrieval query, restricting results to the repositories this specific user is permitted to see. Unauthorized code is never returned, instead of being filtered out afterwards. We also test it with planted marker data in restricted repositories, to confirm nothing leaks across permission boundaries.

Key points

• Per query, not just at login. A user who loses access mid-session must stop getting that repo.
• Pre-filtering, not post-filtering. Retrieve-then-filter has already touched restricted data and wastes result slots.
• Prove it. Plant a canary string in a restricted repo, query as a user without access, confirm it never appears.

Correction

Do not use "ANN" for this. ANN means Approximate Nearest Neighbor, the algorithm used to find similar vectors. The term here is "metadata filter on repo ID". "Scope the search" is fine as shorthand, but unpack it.

ACCESS CONTROL
 X Check once at login           -> weak, can drift
 X Retrieve all, filter after    -> post-filtering, leaks existence
 OK Filter built into the query  -> pre-filtering
    e.g. repo_id IN (user's allowed repos)
 Prove it: canary data in a restricted repo

Answer to give

The first retrieval pass is fast but imprecise, so the right chunk can sit at position 30 of 50. A reranker does a second, more careful pass over just those candidates and pushes the best to the top, so we send only a small number to the model. To know it helps, I run the golden dataset with and without it. If recall stays the same but ranking quality and end-to-end task success improve, it is doing real work. If those numbers do not move, it is not worth the added latency and cost.

Why a separate step

The first search compares vectors and is shallow but fast. A reranker, often a cross-encoder, reads the query and each candidate together, which is more accurate but too slow to run on the whole codebase. It is fine on 50 candidates.

If you did not use one

You said you did not need a reranker. Say that honestly, then show you know the reasoning: you would add one only if measurements showed relevant chunks being found but ranked too low.

RERANKER
 Problem: first pass fast but imprecise
 Fix:     slower second pass on ~50 candidates
 Proof:   A/B on golden set
   recall@50  -> stays the SAME
   ranking quality, task success -> should IMPROVE
   nothing moves -> not worth the latency and cost

You have not practiced this one yet. Here is the answer to learn.

Answer to give

A self-hosted open model is usually weaker than a frontier model at planning multi-step searches and handling very long context. So I design the retrieval layer to do more of the work. The index does the discovery, finding candidate files quickly with hybrid search and ranking, and agentic tools like reading a file or running a test verify the result. I send fewer, more precise chunks, around the top 8 instead of dozens, because noise hurts a smaller model more. I measure on our own golden set whenever we change the model. The benefit is that code never leaves the bank, and the cost is GPU time, so I track tokens per task and cost per resolved task.

Points

• Frontier model means a closed commercial model such as GPT or Claude. An open model is one you host yourself, such as Llama.
• Precision over volume. Effective context on open models is often smaller than the advertised window.
• Retrieval compensates for weaker search planning.
• Re-evaluate on every model change with the golden dataset.

Llama vs vLLM

Llama is a model, the "brain" built by Meta. vLLM is a serving engine, software that runs a model efficiently on your GPUs and handles many requests at once. They work together, they are not alternatives.

OPEN-SOURCE LLM
 Weaker search planning  -> index does discovery, tools verify
 Smaller effective context -> fewer, precise chunks (top 8)
 Self-hosted             -> code stays in bank; cost is GPU time
 Model changes           -> re-run golden dataset

What is a chunk?

We split the repository into chunks, usually one function or class at a time, because splitting by code structure keeps each chunk meaningful on its own. Too small and it loses context. Too big and it wastes tokens and hurts precision.

Why does vector search alone not work for code?

Code needs both semantic search, for conceptual questions, and exact keyword search, for function names and config values that vector search tends to blur together. Using both is hybrid search.

CHUNK = smallest searchable unit of code
  split by structure (function / class), not arbitrary size
VECTOR ALONE FAILS FOR CODE
  semantic -> "how does retry logic work"
  keyword  -> "find ISO8583" or an exact function name
  code needs BOTH = HYBRID SEARCH

Leadership delivery

• Lead with the conclusion, then the reasoning.
• Give a number, even approximate.
• Name the alternative you rejected and why.
• Say the trade-off you accepted.
• End with when you would change the decision.
• Never claim a fact you are unsure of. Flag the uncertainty yourself.